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ABSTRACT 

It  is  increasingly  being  recognized  that  understanding  expert  knowledge  strucb’^s  associated  with  critical 
decision  processes  may  facilitate  Naval  personnel  performance.  Toward  this  J,  system  developers  and 
training  researchers  attempt  to  identify  critical  components  of  expert  operator  assessment  and  knowledge. 
Differing  domains  of  practice  rely  to  varying  degrees  on  perceptual  and  conceptual  knowledge.  Perceptual 
knowledge  is  relied  upon  for  recognizing  critical  cues  in  the  environment  whereas  conceptual  knowledge  is 
used  to  interpret  the  meaning  and  importance  of  these  cues.  We  focus  on  conceptual  knowledge  given  its 
importance  in  the  submariner  environment.  Via  analyses  of  submariner  knowledge  for  concepts  related  to 
responsibilities  for  the  Officer  of  the  Deck  (OOD)  watchstander  we  examined  how  training  may  alter 
knowledge  representation  and  priority  of  conceptual  importance  and  how  overlap  in  mental  models  may  be 
due  to  amount  of  experience. 


INTRODUCTION 

It  is  increasingly  being  recognized  that  understanding 
expert  knowledge  structures  associated  with  critical  decision 
processes  may  facilitate  Naval  personnel  performance  (e.g., 
Gray  &  Kirschenbaum,  2000;  Kirschenbaum,  2001;  Wyman  & 
Randel,  1998).  Toward  this  end,  system  developers  and 
training  researchers  attempt  to  identify  critical  components  of 
expert  operator  assessment  and  decision  processes.  From  the 
standpoint  of  system  development,  such  findings  may  aid  in 
the  identification  of  important  parameters  to  be  included  in 
relevant  operational  environments  (e.g.,  Schvaneveldt, 
Beringer,  &  Lamonica,  2000).  From  the  standpoint  of  training 
and  performance,  targets  for  learning  may  be  identified  (e.g., 
Fiore,  Fowlkes,  Martin-Milham,  &  Oser,  2000).  As  such,  an 
understanding  of  the  knowledge  structures  associated  with 
expert  decision  making  is  critical  to  the  development  of,  not 
only  cognitively  valid  systems,  but  also  efficient  training 
programs. 

Differing  domains  of  practice  rely  to  varying  degrees  on 
perceptual  and  conceptual  knowledge.  Perceptual  knowledge 
is  relied  upon  for  recognizing  critical  cues  in  the  environment 
whereas  conceptual  knowledge  is  used  to  interpret  the 
meaning  and  importance  of  these  cues  (e.g.,  Fiore,  Jentsch, 
Oser,  &  Cannon-Bowers,  2000).  The  issue  centers  on  the  fact 
that  complex  tasks  are  made  up  of  differing  types  of 
knowledge  (e.g.,  declarative,  procedural,  perceptual)  and  the 
acquisition  of  a  particular  knowledge  type  is  likely  to  change 
as  a  trainee  acquires  more  experience  with  a  task.  For 
example,  Lesgold  et  al.  (1988)  noted  that  task  novices  initially 
develop  perceptual  expertise  more  rapidly  than  conceptual 
expertise,  but,  with  experience,  the  conceptual  expertise  gains 
strength. 

Given  its  importance  in  the  submariner  environment,  we 
focus  on  conceptual  knowledge  for  the  following  reasons. 
First,  the  environment  in  which  a  submarine’s  crew  operates 


differs  from  the  normal  environment  in  significant  ways.  The 
submarine  environment  is  a  fluid-filled  medium,  little  or  no 
direct  visual  information  is  provided,  and  much  of  the  acoustic 
information  is  made  available  only  indirectly  and  that,  too,  is 
in  visual  form.  As  a  consequence,  mental  representations  of 
the  submariner’s  world  can  be  greatly  impacted.  Second, 
through  technological  advances,  increasing  amounts  of 
information  are  available  to  the  submarine  watchstander, 
especially  the  Officer  of  the  Deck  (OOD)  (Shobe,  2001).  The 
OOD  controls  the  boat  and  executes  his  mission  by  making 
sense  of  information  presented  by  external  sensors  and 
internal  processing.  The  OOD  uses  this  information  to  build 
his  mental  model  and,  in  this  study,  we  examined  knowledge 
representation  within  experienced  submariner  officers  in  order 
to  determine  how  such  knowledge  may  change  with 
experience. 

We  next  describe  our  analyses  of  submariner  knowledge 
for  concepts  related  to  responsibilities  for  the  OOD  task.  We 
examined  how  training  may  alter  knowledge  representation, 
and  how  overlap  in  mental  models  may  be  due  to  amount  of 
experience.  Similar  research  has  assessed  the  nature  and 
sharedness  of  mental  models  for  submariners.  For  example, 
Smith-Jentsch  et  al.  (2001)  looked  at  how  experience 
influenced  mental  models  of  teamwork  and  found  greater 
agreement  based  upon  either  rank  or  years  in  service. 
Additionally,  when  compared  to  an  expert  model,  those  with  a 
higher  rank  showed  greater  agreement.  We  expand  upon  that 
research  to  examine  how  mental  models  associated  with  the 
OOD  task  may  show  similar  patterns  and  further  investigate 
how  information  priority  may  change  due  to  experience. 
Priority  rankings  data  for  task  related  concepts  may  elucidate 
differences  in  conceptual  knowledge  across  groups  varying  in 
experience.  For  example,  in  a  study  looking  at  phases  of 
flight,  Schvaneveldt,  Beringer,  and  Lamonica  (2000)  found 
that  this  technique  was  able  to  highlight  differences  in 
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perceptions  of  importance  dependent  upon  the  level  of 
expertise  of  the  pilots. 

Based  upon  shared  mental  model  theory  (e.g.,  Cannon- 
Bowers,  Converse,  &  Salas,  1993;  Fiore,  Salas,  &  Cannon- 
Bowers,  2001),  we  hypothesized  that,  first,  the  experienced 
participants  would  show  greater  similarity  within  their  groups 
and  to  an  expert  model.  Second,  following  training,  novice 
participants  would  (a)  show  greater  agreement  in  mental 
models  for  the  critical  concepts,  and  (b)  show  greater 
agreement  with  expert  participants  (i.e.,  a  composite  model 
based  upon  the  input  of  a  group  of  expert  participants). 
Additionally,  rankings  of  importance  for  these  critical 
concepts  were  expected  to  show  how  novice  and  less 
experienced  personnel  may  misunderstand  priorities 
associated  with  decision-making  criteria  when  compared  to 
experts. 

METHOD 

Participants 

Experienced  participants  ( N  =  40)  were  qualified  Officers 
of  the  Deck  (OOD)  in  Submarine  Officer  Advanced  Course 
(SOAC).  Novice  participants  ( N  =  50)  were  students  in  their 
first  week  of  Submarine  Officer  Basic  Course  (SOBCpre)  with 
no  OOD  experience.  These  same  participants  were  tested 
again  at  the  end  of  the  12  week  course  (SOBCpst).  The  SOBC 
group  is  not  OOD  qualified,  but  after  completion  of  the  course 
they  have  had  all  the  relevant  lectures  and  trainer  experience. 
Years  in  Service  for  participants  in  the  SOBC  group  (M  =  5.12 
years)  was  significantly  lower  than  that  of  participants  in  the 
SOAC  group  (M  =  10.58  years),  F  (1,  89)  =  34.76,  p  <  .001. 
Years  in  Present  Rank  for  participants  in  the  SOBC  group  (M 
=  1.6  years)  was  significantly  lower  than  that  of  participants  in 
the  SOAC  group  ( M  =  3.3  years),  F  (1,  88)  =  62.66,  p  <  .001. 
Years  Sea  Duty  for  participants  in  the  SOBC  group  (M  =  1.1 
years)  was  significantly  lower  than  that  of  participants  in  the 
SOAC  group  ( M  =  3.8  years),  F  (1,  89)  =  35.92,  p  <  .001. 
Data  from  a  small  group  of  experts  ( N  -  4)  was  also  collected 
in  order  to  develop  an  expert  composite  model  for  comparison 
to  the  more  and  less  experienced  participants.  The  expert 
participants  were  all  Lieutenant  Commanders  and  overall,  had 
an  average  length  of  8.9  years  in  that  rank.  The  mean  number 
of  years  in  service  was  20.4,  with  an  average  of  6.5  years 
submarine  sea  duty.  All  of  the  expert  participants  had  served 
as  department  heads  on  submarines  and  were  currently  filling 
a  Post-Department  Head  shore  tour  billet. 

Materials 

Critical  categories  of  information  available  to  the 
submarine  OOD  were  printed  separately  on  20  index  cards. 
Each  of  the  20  cards  contained  a  piece  of  information 
available  at  CONN  for  navigation  and  ship  safety.  Tactical 
and  weapons  information  was  not  provided. 

Procedure 

Participants  performed  both  a  card  sorting  and  ranking 
task  to  indicate  similarity  and  relative  importance  among 
categories  of  information  available  on  submarines  in  different 


operational  environments.  Participants  were  instructed  to  sort 
the  cards  into  piles  according  to  similarity  and  were  not 
constrained  by  the  number  of  piles  they  could  create. 
Following  the  card  sorting  task,  participants  ranked  the  cards 
according  to  relative  importance.  They  were  instructed  to 
create  a  single  pile  of  cards  ordered  from  the  most  important 
piece  of  submarine  information  to  the  least  important. 

Design 

This  experinv“nf  manipulated  experience  level 
(experienced  vs.  nov  )  as  a  between  participant  factor.  For 
the  concept  rankings  measure,  operational  scenario  was 
manipulated  via  factorial  combination  of  presence/absence  of 
enemy  and  shallow/deep  water. 

RESULTS 

For  the  card  sort  data,  in  order  to  assess  the  relation 
among  concept  pairs,  each  possible  concept  pair  (N  =  210) 
was  coded  with  a  0  if  the  participant  did  not  group  them  in  the 
same  category,  or  a  1  if  they  were  grouped  in  the  same 
category.  For  the  concept  rankings  data,  mean  concept 
importance  was  calculated  for  each  concept  across  groups  and 
scenarios. 

Mental  Model  Similarity  for  Card  Sort  Data 

Overall  Correlations.  This  data  determines  the  degree  to 
which  participants  may  differ  in  their  mental  models  of  critical 
OOD  concepts.  Using  the  concept  pair  coding  scheme 
described  above  we  computed  correlations  with  all  possible 
participant  pairings.  These  cross-correlations  were  run  across 
and  within  the  three  participant  groups  (SOAC,  SOBCpre, 
SOBCpst).  The  data  based  upon  this  matrix  thus  had  six 
conceptual  groups:  (1)  SOAC  correlated  with  SOAC;  (2) 
SOAC  correlated  with  SOBCpst;  (3)  SOAC  correlated  with 
SOBCpre;  (4)  SOBCpre  correlated  with  SOBCpst;  (5) 
SOBCpst  correlated  with  SOBCpst;  and,  (6)  SOBCpre 
correlated  with  SOBCpre.  This  data  was  subjected  to  a 
Univariate  ANOVA  and  resulted  in  a  significant  main  effect 
for  group  type,  F  (5,  8250)  =  20.95,  p  <  .001.  For  ease  of 
explication  we  present  this  data  as  mean  within  group 
correlations  and  mean  across  group  correlations.  Specifically, 
the  within  group  correlations  corresponded  to  “SOAC 
correlated  with  SOAC”,  “SOBCpst  correlated  with  SOBCpst”, 
and  “SOBCpre  correlated  with  SOBCpre”.  The  across  group 
correlations  corresponded  to  “SOAC  correlated  with 
SOBCpst”,  “SOAC  correlated  with  SOBCpre”,  and 
“SOBCpre  correlated  with  SOBCpst”  (refer  to  Figure  1). 
Unless  otherwise  specified,  the  reported  post-hoc  analyses  are 
all  significant  at  the  p  <  .05  level. 

When  considering  the  within  group  comparisons,  post-hoc 
analyses  showed  that  the  degree  of  mental  model  similarity 
was  significantly  greater  for  the  more  experienced  (SOAC) 
participants  (M  =  .39).  The  degree  of  mental  model  similarity 
was  significantly  lower  for  the  least  experienced  (SOBCpre) 
participants  ( M  =  .33),  with  mental  model  similarity  for 
SOBCpst  in  between,  and  significantly  different  from,  the 
other  groups  ( M  =  .35).  When  considering  the  across  group 
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comparisons,  post-hoc  analyses  showed  that  the  degree  of 
across  group  mental  model  similarity  was  greatest  for  SOAC 
and  SOBCpst  ( M  =  .36).  Correlations  between  SOBCpre  and 
SOBCpst  ( M  -  .33)  and  between  SOAC  and  SOBCpre  ( M  = 
.33)  were  both  significantly  lower  than  the  correlation  between 
SOAC  and  SOBCpst. 
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Figure  1.  Mean  correlations  across  all  possible  participant 
pairings  resulting  from  card  sort  data. 

Similarity  to  Expert  Composite  Model.  We  additionally 
examined  the  degree  to  which  these  participant  groups 
generated  mental  models  similar  to  a  composite  expert  model. 
We  were  interested  in  assessing  whether  training  and/or 
experience  influenced  structural  similarity  to  an  expert  model 
and  we  hypothesized  that  the  differing  groups  would  show 
differing  levels  of  similarity  to  the  expert  model.  For  this 
analysis,  we  computed  mean  correlations  with  the  expert 
composite  model  for  each  participant  group.  An  ANOVA 
revealed  a  main  effect  of  group  F  (2,  126)  =  19.76,  p  <  .001. 
Post-hoc  analyses  showed  that  the  SOAC  participants  ( M  = 
.43)  were  significantly  higher  in  agreement  to  the  expert 
model  than  were  the  SOBCpst  participants  ( M  =  .35),  who 
were,  in  turn,  significantly  higher  than  the  SOBCpre 
participants  (M=  .28). 

Concept  Rankings  of  Importance 

We  next  discuss  a  portion  of  the  results  pertaining  to  our 
analyses  of  the  degree  to  which  trainees  view  concept 
importance  differently.  Multi-dimensional  scaling  on  the  card 
sort  data  for  the  composite  expert  group  was  used  to  identify 
clusters  (reported  in  Shobe,  2002).  These  clusters  served  as  an 
organizing  framework  with  which  to  analyze  and  interpret  the 
differences  in  importance  ratings  and  more  specifically  target 
problematic  concepts  within  the  overall  knowledge  structure. 
Due  to  page  limitations,  we  only  report  the  overall  effects  for 
the  rankings  data  and  the  comparisons  to  the  expert  group  and 
only  for  a  portion  of  the  clusters.  For  illustrative  purposes, 
these  concepts  are  graphed  according  to  a  difference  score 


calculated  using  the  mean  expert  score.  A  negative  score 
means  participants  within  a  given  group,  on  average,  viewed 
this  concept  as  less  important  than  the  expert  group. 

Own  Ship  Parameters  and  Contact  Management.  Table  1 
shows  the  concepts  for  the  “Own  Ship  Parameters  and  Contact 
Management”  cluster.  The  mean  concept  importance  ratings 
for  the  four  groups  (SOAC,  SOBCpre,  SOBCpst,  Experts)  was 
subjected  to  a  Multivariate  ANOVA.  Group  type  was  the 
independent  factor  aH  *he  seven  concepts  identified  in  Table 
1  were  the  depende...  variables.  There  was  a  significant 
multivariate  main  effect  for  group  type,  F  (21,  1560)  =6.51  ,p 
<  .001.  For  ease  of  explication,  Table  2  presents  the  between 
subjects  effects  for  this  analysis.  Similarly,  Figure  2  illustrates 
the  pattern  of  differences  for  the  effect  of  group  type  on  mean 
concept  rankings  for  the  Own  Ship  Parameters  and  Contact 
Management  cluster.  Significant  differences  between  the 
Expert  group  and  the  participant  groups  are  noted  in  the 
footnotes  for  Table  2. 

Table  1 

Concept  Names  and  Explanations  for  the  Own  Ship 
Parameters  and  Contact  Management  Cluster 


Cluster  One: 

Own  Ship  Parameters  and  Contact  Management 

Concept 

Description 

1 

ESM  contact  data 

Bearing  and  classification  of  radar 
and  communications  systems  data 
provided  by  the  Electronic  Support 
Measures  system 

2 

Own  ship  data 

Own  ship’s  course,  speed,  depth,  etc. 

8 

Visual  contact  data 

Visual  information  on  surface 

(from  photonics 

contacts  including  night  vision, 

mast) 

infrared,  video,  laser  ranging  etc. 

14 

Geosit/ops 

Computer  generated  geographical 

summary 

picture  of  contacts  and  own  ship,  in 
either  true  or  relative  bearing 
orientation,  with  classification 
information  where  possible 

15 

Trial  own  ship 

CPA  solutions,  trial  maneuvers,  etc., 
i.e.,  data  which  aid  in  assessment  of 
the  present  and  future  tactical 
situation 

16 

Non-target  fire 

Rapid  passive  localization  by  wide 

control  solution 

aperture  array  (WAA),  KAST 
ranging,  multipath  ranging,  D/E 
ranging,  hyperbolic  ranging;  Bearing, 
range,  course,  speed,  depth  if 
submerged  contact,  for  secondary 
contacts 

19 

Target  fire  control 

Rapid  passive  localization  by  wide 

solution 

aperture  array  (WAA),  KAST 
ranging,  multipath  ranging,  D/E 
ranging,  hyperbolic  ranging;  Bearing, 
range,  course,  speed,  depth  if 
submerged  contact,  for  target  of 
interest 
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Table  2 

Results  from  Multivariate  Analysis  for  Own  Ship  Parameters 
and  Contact  Management  Cluster 


Source 

DV 

Type  III 

df 

Mean 

F 

Sig. 

SS 

Square 

Participant  C01  a 

803.098 

3 

267.699 

10.359 

.000 

Group 

C02e 

61.430 

3 

20.477 

1.274 

.283 

C08  a 

1813.450 

3 

604.483 

18.289 

.000 

C14c 

466.602 

3 

155.534 

9.513 

.000 

C15  e 

302.838 

3 

100.946 

5.417 

.001 

C16b 

410.182 

3 

136.727 

7.004 

.000 

C19d 

368.573 

3 

122.858 

5.398 

.001 

Error 

C01 

13540.902 

524 

25.841 

C08 

17319.459 

524 

33.052 

C14 

8566.914 

524 

16.349 

C15 

9764.677 

524 

18.635 

C16 

10228.477 

524 

19.520 

C19 

11925.379 

524 

22.758 

Notes: 

a  SOBCpre,  SOBCpst,  and  SOAC  significantly  different  from 
Expert  Group 

b  SOBCpre  and  SOBCpst  significantly  different  from  Expert 
Group 

c  SOBCpst  significantly  different  from  Expert  Group 
d  SOAC  significantly  different  from  Expert  Group 
None  significantly  different  from  Expert  Group 


El  SOBCpre 
■  SOBCpost 
□  SOAC 


Trial  Own  Target  Fire 

Ship  Control  Solution 

Non-target  Fire 
Control  Soluion 


and  the  four  concepts  identified  in  Table  3  were  the  dependent 
variables.  There  was  a  significant  multivariate  main  effect  for 
group  type,  F  (12,  1569)  =  12.06,  p  <  .001.  For  ease  of 
explication,  Table  4  presents  the  between  subjects  effects  for 
this  analysis.  Similarly,  Figure  3  illustrates  the  pattern  of 
differences  for  the  effect  of  group  type  on  mean  concept 
rankings  for  the  Sonar  Tactical  Displays  cluster.  Significant 
differences  between  the  Expert  group  and  the  participant 
groups  are  noted  in  the  footnotes  for  Table  4. 

Table  3 

Concept  Names  and  Explanations  for  the  Sonar  Tactical 
Displays  Cluster 


Cluster  Four: 

Sonar  Tactical  Displays 

Concept 

Description 

3 

PMFL  data 

Data  on  fault  location  and  performance 
monitoring  of  various  sonar 
components,  i.e.,  data  required  to 
maintain  electrical  information 
systems 

12 

Sonar 

Auditory  presentation  of  minimally 

tracker/cursor 

processed  sonar  signals  as  picked  up 

audio 

by  the  various  arrays 

17 

Sonar 

Visual  displays  of  sonar  detections  as 

detection 

displays 

presented  to  sonar  operators 

20 

Sonar  class 

Sonar  signal  interpretation  aids  such  as 

displays 

signature  assemblies,  lofargrams,  etc. 
i.e.,  data  used  to  aid  in  the 
classification  of  signals 

Table  4 

Results  from  Multivariate  Analysis  for  Sonar  Tactical 
Displays  Cluster 


Source 

DV 

Type  III 

df 

Mean 

F 

Sig. 

SS 

Square 

Participant 

C03a 

451.424 

3 

150.475 

12.091 

.000 

Group 

C12a 

97.034 

3 

32.345 

1.634 

.180 

C17b 

2464.951 

3 

821.650 

44.870  .000 

C20c 

421.197 

3 

140.399 

6.511 

.000 

Error 

C03 

6521.256 

524 

12.445 

C12 

10370.959 

524 

19.792 

C17 

9595.292 

524 

18.312 

C20 

11298.864 

524 

21.563 

Concepts  for  Own  Ship  Parameters 

Figure  2.  Difference  scores  for  Own  Ship  Parameters  and 
Contact  Management  cluster. 

Sonar  Tactical  Displays.  Table  3  lists  and  defines  the 
concepts  making  up  the  “Sonar  Tactical  Displays”  cluster. 
The  mean  concept  importance  ratings  for  the  four  groups 
(SOAC,  SOBCpre,  SOBCpst,  Experts)  was  subjected  to  a 
Multivariate  ANOVA.  Group  type  was  the  independent  factor 


Notes: 

a  None  significantly  different  from  Expert  Group 
SOBCpre  and  SOBCpst  significantly  different  from  Expert 
Group 

c  SOBCpre,  SOBCpst,  and  SOAC  significantly  different  from 
Expert  Group 
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B  SOBCpre 
■  SOBCpost 
□  SOAC 


Sonar  Tracker/  Sonar  Detection  Sonar  Class 
Cursor  Audio  Displays  Displays 


Concepts  (or  Sonar  Tactical  Displays 


Figure  3.  Difference  scores  for  Sonar  Tactical  Displays 
Cluster. 

DISCUSSION 

In  support  of  shared  mental  model  theory  and  consistent 
with  other  studies  of  knowledge  organization  and  the  mental 
models  of  submariners  (e.g.,  Smith-Jentsch  et  al.,  2001),  this 
investigation  found  that  more  experienced  personnel  viewed 
conceptual  linkages  more  similarly  and  showed  higher 
agreement  with  an  expert  model  (Fiore,  Fowlkes,  Martin- 
Milham,  &  Oser„  2000).  Conversely,  the  least  experienced 
personnel  saw  conceptual  linkages  less  similarly,  but, 
following  training,  these  participants  were  found  to  view  the 
conceptual  linkages  more  similarly  to  the  experienced 
personnel.  From  the  standpoint  of  understanding  the 
importance  of  particular  concepts,  significant  differences 
between  experienced  and  novice  personnel  were  identified 
prior  to,  and  following,  training.  This  data  highlight  how 
participants  vary  in  their  understanding  of  the  importance  of 
these  concepts. 

Linking  the  conceptual  groupings  with  the  priorities  data 
can  better  illuminate  knowledge  organization  and  the  differing 
ways  that  knowledge  may  be  tapped  in  operational 
environments  (cf.  Schvaneveldt  et  al.,  2001).  From  the 
standpoint  of  training,  this  data  suggest  that  the  conceptual 
grouping  garnered  from  the  expert  group  may  aid  in 
conveying  important  clusters  of  information  as  well  as  how 
priorities  vary  within  these  clusters.  This  data  is  theoretically 
important  because,  from  the  mental  model  standpoint,  it 
distinguishes  more  and  less  experienced  personnel  on  the  basis 
of  critical  misconceptions  in  item  importance.  This  data  is 
practically  important  because  these  misconceptions  can  be 
considered  as  targets  for  training.  For  example,  this  allows 
one  to  identify  over-  and  under-statement  of  importance  for 
trainees  at  different  levels.  This,  in  turn,  suggests  what 
concepts  need  augmentation  in  the  training  (i.e.,  where 
misconceptions  in  concept  importance  lie  before  and  after 
training)  and  may  help  with  redesign  of  systems  based  upon 
the  importance  of  data. 
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